1. Technical Field
The invention is related to deghosting mosaiced images, and more particularly to a system and process for deghosting a mosaiced image of a scene produced by stitching together overlapping images of the scene, each of which was captured by a different camera from a different viewpoint.
2. Background Art
Typically, mosaiced images, such as a panorama mosaic, are created by stitching together images taken by a single camera at a fixed viewpoint that is rotated to capture images of the surrounding scene. However, more recently, such mosaics, and particularly panoramic mosaics, have been created using a series of cameras each capturing a different part of the surrounding scene from a different viewpoint. This type of system for capturing images and creating mosaiced images by stitching the corresponding images captured by the respective cameras has been employed in the authoring of panoramic videos. In such a system, the cameras are video cameras and the images captured by the cameras at the same moment in time are stitched together to form a frame of the panoramic video.
A particular problem associated with stitching images together that were captured by different cameras at different viewpoints is localized double images of objects in the scene, which is often referred to as parallax distortion or ghosting. While it is irrelevant for objects in the scene that are far away from the cameras, objects that are closer in can create a double image in the mosaiced image. It is akin to a person bringing an object closer and closer to their eyes. At some point the object appears as two instead of one because the person""s eyes are viewing it from different viewpoints. In the context of a mosaiced image, a close-in object depicted in two of the images captured at different viewpoints will result in a double image in the overlap region of these images in mosaiced image. This problem can be particularly acute in the creation of a panoramic video as objects will typically move in and out of the scene and may come close enough to the camera rig to cause the aforementioned localized double image. Up to this time no provision has been made to compensate for this double imaging problem, with the exception of eliminating any close in objects in the scene. The present deghosting technique addresses this problem without necessitating the elimination of close in objects.
The present invention is directed toward a system and process for deghosting mosaiced images created by stitching together images of a scene captured from different viewpoints. As discussed above, when images are stitched together (i.e., mosaiced) that were captured by different cameras at different viewpoints, the possibility of localized double images of objects exists. Essentially, this double imaging or ghosting will occur if an object in the scene is close in to the cameras capturing the images. However, in accordance with the present invention this localized ghosting can be compensated for by estimating the amount of local mis-registration and then locally warping each image in the mosaiced image to reduce any ghosting. Preferably, this is accomplished using the procedures described in U.S. Pat. No. 5,986,668 entitled xe2x80x9cDeghosting Method and Apparatus for Construction of Image Mosaicsxe2x80x9d, the disclosure of which is hereby incorporated by reference. This method was developed to reduce distortions caused by the fact that typical mosaicing processes assume an idealized camera model. However, in actuality un-modeled radial distortion (i.e., that component of the radial distortion that cannot be adequately modeled in the mosaicing process), tangential distortion, and non-square pixel distortion, among others can cause the local mis-registrations which can be corrected for using the aforementioned method. The same procedures employed to reduce these types of distortion can be adapted to reduce ghosting as well. Essentially, the deghosting process involves computing the optical flow between all pairs of images, and inferring the desired local warps from these computations.
The foregoing deghosting technique is particularly useful in the context of generating panoramic video using a non-catadioptric, multi-camera panoramic video authoring system, when employed on a frame by frame basis. As mentioned previously, these types of authoring systems use a series of video cameras to capture different parts of the surrounding scene from different viewpoints. In such a system, the images captured by the cameras at the same moment in time are stitched together to form a frame of the panoramic video. However, objects will typically move in and out of the scene and may come close enough to the camera rig to cause the aforementioned localized double image. Given this, each mosaiced frame of the panoramic video would be deghosted in accordance with the present technique to reduce the localized mis-alignments that are responsible for the ghosting.
In addition to the just described benefits, other advantages of the present invention will become apparent from the detailed description which follows hereinafter when taken in conjunction with the drawing figures which accompany it.